Skip to content

slurm script inference#1675

Merged
grassesi merged 8 commits into
ecmwf:developfrom
mtar:1308-slurm-script-inference
Jan 26, 2026
Merged

slurm script inference#1675
grassesi merged 8 commits into
ecmwf:developfrom
mtar:1308-slurm-script-inference

Conversation

@mtar

@mtar mtar commented Jan 22, 2026

Copy link
Copy Markdown
Contributor

Description

Decide on the training stage based on the first argument. Added part to !99

Issue Number

Closes #1308

Is this PR a draft? Mark it as draft.

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

@mtar mtar changed the title 1308 slurm script inference slurm script inference Jan 22, 2026
Comment on lines -193 to +202
# Entry point for slurm script.
# Check whether --from_run_id passed as argument.
if any("--from_run_id" in arg for arg in sys.argv):
train_continue()

if any("train" in arg for arg in sys.argv):
# Entry point for slurm script.
# Check whether --from_run_id passed as argument.
if any("--from_run_id" in arg for arg in sys.argv):
train_continue()
else:
train()
elif any("inference" in arg for arg in sys.argv):
inference()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just adding a "train" or "inference" as an argument breaks the cli parser since it does not know about these arguments yet. Please pull the changes from mtar#1, or implement the stage argument properly in cli.py

@github-project-automation github-project-automation Bot moved this to In Progress in WeatherGen-dev Jan 26, 2026
@grassesi
grassesi marked this pull request as ready for review January 26, 2026 01:55
get stage info from environment variable.
@grassesi
grassesi merged commit 0c7822d into ecmwf:develop Jan 26, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in WeatherGen-dev Jan 26, 2026
iluise pushed a commit that referenced this pull request Jan 27, 2026
* add argument

* check stage argument

* removed unnecessary code

* arbitrary position arguments

* Fix error text

* get stage info from environment variable.

* Update run_train.py

---------

Co-authored-by: Simon Grasse <s.grasse@fz-juelich.de>
@tjhunter tjhunter mentioned this pull request Feb 4, 2026
6 tasks
@mtar
mtar deleted the 1308-slurm-script-inference branch March 11, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Slurm scripts for inference

2 participants